Can this be promoted to actual generator by including _sample_sequences and _sequences_to_circuits? I am implementing another RB variant, that partly uses Clifford sequence but final structure is totally different from standard RB (thus no inheritance). I don't want to implement boilerplate code in such a class.

I see your point, but let me go there (having RBCircuitGenerator class to delegate whole circuit generation) step by step. I'd like to focus on clean-up of Standard/InterleavedRB implementation in this PR, then clean-up of clifford_utils in the next PR (I'll remove the creation of CliffordUtils objects there). I prefer further refactoring should come after them.

I realized current implementation induces a serious issue when benchmarking the user-calibrated gate. Since conventional sequence implicitly triggers the pulse gate pass in the default pass managers that pulls gate calibrations from the backend instmap that doesn't explicitly show up in the transpile option. This is indeed the breaking API change, because custom calibration will be just ignored but user will never notice this. I think you need to explicitly manage inst map without using pass manager (for performance reason). Alternatively, we can assume V2 backend, where calibration data is always provided with gate entry. Note that this PR will be merged soon. This makes me think we need unittest for missing calibrations.

Addressed in cee4e78

I prefer to have warning or completely remove transpile options, i.e overriding _set_transpile_options method to raise warning/error. Usually users are not aware of what is happening inside .run() and may set some random option from random example code. I think having inst_map only makes sense in this experiment.

Good suggestion. I like the idea restricting acceptable transpile options for each experiment. (too many possible transpile_options was one of my headaches.) For RB experiments, I don't like to but I'll accept basis_gates and optimization_level for now as well as inst_map. Because they are widely used in the tests (Fortunately, they seem not to be used in the tutorial). Forbidding them would require large refactoring of tests, so if we want that, I think it should be done in a separate PR.

Do you think worth having multi-threading here? Note that the standard Qiskit transpiler uses qiskit parallel and this is single thread.

Yeah, I think it's worth investigating.

I've examined parallel transpilation using qiskit.tools.parallel.parallel_map and found that it's much slower than serial one. I guess that is due to the overhead of pickling circuits. Let me keep the current code as is.

Examined code for parallel transpilation:

            circuits = self.circuits()
            if (
                    len(circuits) > 1
                    and os.getenv("QISKIT_IN_PARALLEL", "FALSE") == "FALSE"
                    and parallel.PARALLEL_DEFAULT
            ):
                # Custom transpilation in parallel
                transpiled = parallel.parallel_map(
                    _transpile_clifford_circuit,
                    circuits,
                    task_kwargs={"physical_qubits": self.physical_qubits},
                )

This is good to know.

Or maybe because I'm using Mac (I somewhere heard parallel does multi-processing so slow in Mac)

On mac by default parallel_map won't actually run in parallel on Python >= 3.8 (https://github.com/Qiskit/qiskit-terra/blob/main/qiskit/tools/parallel.py#L61-L83 defines the defaults) because in Python 3.8 they switched macOS multiprocessing to use spawn instead of fork by default because of some platform bugs. Spawn has a lot of extra overhead and it's also unreliable for user code because you need a if __name__ == "__main__" guard many times to distinguish a child process from the parent so we disable multiprocessing by default on macOS.

In general multiprocessing in python can be very tricky to work with and get a performance boost. Although for transpile() with linux (so using fork()) with involved compilation the performance is typically better even with the pickling and IPC overhead of separate processes (with some caveats Qiskit/qiskit#7741 )

Do you need full transpile here? Likely you just need to call basis translator.

Good point. I think it was for delays but it may be unnecessary now since I updated the handling of delays in 36193c9
Let me check if we can replace transpile with basis translation.

I found it's possible but it would be difficult to explain the spec in doc (since we'd need to mention to a protected method _transform_clifford_circuit). Now I think we should revisit here when we redesign the basis translation scheme (removing the use of transpile for 1Q/2Q RB) in the future.

This comment is not addressed. Of course you can do this in the follow-up.